🌻 Strengthening OH with causal mapping
PDF

Strengthening OH with causal mapping#

image.png

Here is version of the final text.

Strengthening Outcome Harvesting Analysis with AI-Assisted Causal Mapping - shortened full version#

Written by: Heather Britt, Steve Powell, Gabriele Caldas Cabral

Summary#

This case study explores how AI-assisted causal mapping can enhance Outcome Harvesting (OH) analysis by revealing interrelationships between outcomes and identifying new actors contributing to change. The pilot demonstrates how this approach provides actionable insights and strengthens causal relationship analysis in OH. It emphasizes the importance of a principle-led analysis plan and human expertise in guiding the AI process.

Introduction#

Outcome Harvesting is a powerful approach for discovering emergent changes—whether predicted or unpredicted, positive or negative—and documenting how those changes occurred. While many methods capture changes in those directly involved with a project, OH captures changes farther down the causal pathway.

However, evaluators often struggle to explore interrelationships between multiple outcomes. This case study describes how an OH practitioner (Heather Britt) collaborated with causal mapping practitioners (Steve Powell and Gabriele Caldas) to expand causal contribution analysis in OH using an AI-assisted causal mapping app. They analyzed OH data from a completed education project.

Outcome Harvesting: Analysis Limitations#

While OH documents causal pathways contributing to individual outcomes well, evaluators find it difficult to make sense of interrelationships between multiple outcomes and their causal pathways. This limits their ability to answer questions about causal contribution.

Current OH practice often uses descriptive statistics to summarize data by outcome components (e.g., types of change agents or social actors) and reports findings in charts. Another approach arranges outcomes on a timeline to determine logical relationships.

Our pilot explores whether AI-assisted causal mapping can address these limitations by analyzing causal relationships between outcomes.

The Pilot#

Core Question#

Can AI-assisted causal mapping address the limitations of OH analysis?

Heather Britt reached out to Steve Powell and Gabriele Caldas to explore whether causal mapping with the Causal Map app could enhance OH analysis.

Causal mapping techniques, developed over 50 years ago, have been used across disciplines to identify and visually represent causal relationships in qualitative data. The Causal Map app computerizes this technique, allowing efficient coding, analysis, and visualization of information from multiple sources (interviews, reports, surveys, narratives), either manually or with AI assistance.

The AI-assisted capacities of the app were critical for revealing interrelationships between multiple outcomes.

Pilot Data Set#

The pilot used data from the final evaluation of an education project (Girls Education project, 2016–2021) disrupted by political turmoil and COVID-19. The project adapted activities during lockdown, and OH was used to capture outcomes in five domains where the theory of change was no longer valid.

The evaluation team interviewed 49 change agents and drafted 103 outcome descriptions across five domains. The pilot data included both interview transcripts and outcome descriptions.

For the pilot, the domain Increased community support for education was selected, with 13 outcome descriptions analyzed.

Analysis Process#

Step 1: Draft a Principle-Led Analysis Plan#

Three guiding principles steered analysis decisions:

  1. Prioritize local leadership: Use AI while keeping sensemaking and learning in the hands of local evaluators.
  2. Protect OH integrity: Adapt methods as needed while staying true to OH principles, including “Less is more” (avoid collecting more data than can be analyzed).
  3. Produce accurate, actionable maps: Human judgment is required to error-check data and interpret maps.

Step 2: Segment Data by Outcome Domain#

Segmenting data by domain increases the likelihood of finding coherent causal pathways and facilitates error-checking. The pilot focused on one domain to analyze causal relationships between outcomes.

Step 3: Decide When to Apply AI-Assisted Causal Mapping#

The pilot compared applying causal mapping to interview transcripts versus outcome descriptions. Outcome descriptions, crafted by local evaluators, were more accurate and required less error-checking than transcripts. Thus, mapping outcome descriptions was preferred to preserve local leadership and OH integrity.

Findings from Causal Maps#

Relationships Between Outcomes#

The AI identified causal links between the 13 outcome descriptions, revealing that outcomes influenced one another. For example, parents actively supporting home learning and leaders convincing parents to participate were central factors.

Factors Contributing to Domain-Level Outcome#

Mapping revealed additional actors influencing the domain-level outcome “Community supports learning,” including unexpected contributors like Ministry officials.

Conclusion#

AI-assisted causal mapping advanced OH analysis beyond descriptive statistics by:

Causal mapping offers rich, flexible analysis that can be explored in multiple ways to answer diverse evaluation questions.